feat: keep a hint of last snapshot location while committing new snapshot #7418
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I hereby agree to the terms of the CLA available at: https://databend.rs/dev/policies/cla/
Summary
Background:
this pr is a sub aux task of database object sharing.
By keeping a hint file for each table, which contains the location of the table's last snapshot, the
sharing service
could work by accessing object storage only (without touching metastore service or databend-query nodes).@ZhiHanZ @flaneur2020 please correct me if the above summary is not right.
some refs:
Functionality of this PR
whenever a table has successfully committed, try keeping a hint file that indicates the location of the last snapshot of given table:
/$STORAGE_PREFIX/$DB_ID/$TABLE_ID/last_snapshot_location_hint
/$STORAGE_PREFIX/$DB_ID/$TABLE_ID/xxxxx.json
where
STORAGE_PREFIX
is configured by the parameterroot
of storage typese.g. for local file system storage type, the hint looks like:
/tmp/.tmpNP66QH/1/11/last_snapshot_location_hint
/tmp/.tmpNP66QH/1/11/_ss/28a2109a67544c73a5af922acbabce68_v1.json
NOTE:
misc:
Fixes #issue